Skip to content

[SPARK-12046] [DOC] [1.6] ScalaDoc and JavaDoc fixes#10039

Closed
liancheng wants to merge 1 commit into
apache:branch-1.6from
liancheng:spark-12046.doc-fix
Closed

[SPARK-12046] [DOC] [1.6] ScalaDoc and JavaDoc fixes#10039
liancheng wants to merge 1 commit into
apache:branch-1.6from
liancheng:spark-12046.doc-fix

Conversation

@liancheng

Copy link
Copy Markdown
Contributor

This PR fixes several issues in our Scala and Java API doc.

  • Hides spark.memory package

  • Hides spark.examples package

  • Fixes ScalaDoc unordered list format through out the code base

    Unordoreed list in ScalaDoc requires an extra space before - or *, i.e.:

    /**
      * A list:
      *
      *  - Item 1
      *  - Item 2
      */

    rather than

    /**
      * A list:
      *
      * - Item 1
      * - Item 2
      */

    Otherwise the list won't be properly rendered.

  • Some other minor formatting issues

Remaining issues that I haven't figured out a proper fix:

  • :: DeveloperApi :: badge is not rendered properly on my local machine, but not sure whether it's an environmental issue.

  • akka package still appears in ScalaDoc root package page, even though it's already filtered in SparkBuild.scala.

  • Seems that most stuff in spark.util shouldn't be public, e.g.

    • RpcUtils
    • InnerClosureFinder

    But they've been there for a long time. Not sure whether they are already considered as part of the public API?

  • KinesisUtilsPythonHelper shows up in JavaDoc even though it's marked as private

@liancheng

Copy link
Copy Markdown
Contributor Author

cc @rxin

@srowen

srowen commented Nov 30, 2015

Copy link
Copy Markdown
Member

LGTM

@rxin

rxin commented Nov 30, 2015

Copy link
Copy Markdown
Contributor

LGTM too

@SparkQA

SparkQA commented Nov 30, 2015

Copy link
Copy Markdown

Test build #46878 has finished for PR 10039 at commit 81d973b.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai

yhuai commented Nov 30, 2015

Copy link
Copy Markdown
Contributor

test this please

@SparkQA

SparkQA commented Nov 30, 2015

Copy link
Copy Markdown

Test build #46896 has finished for PR 10039 at commit 81d973b.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing

zsxwing commented Dec 1, 2015

Copy link
Copy Markdown
Member

retest this please

asfgit pushed a commit that referenced this pull request Dec 1, 2015
KinesisStreamTests in test.py is broken because of #9403. See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46896/testReport/(root)/KinesisStreamTests/test_kinesis_stream/

Because Streaming Python didn’t work when merging #9403, the PR build didn’t report the Python test failure actually.

This PR just disabled the test to unblock #10039

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #10047 from zsxwing/disable-python-kinesis-test.
@liancheng

Copy link
Copy Markdown
Contributor Author

retest this please

@yhuai

yhuai commented Dec 1, 2015

Copy link
Copy Markdown
Contributor

test this please

@yhuai

yhuai commented Dec 1, 2015

Copy link
Copy Markdown
Contributor

should we also update master?

@JoshRosen

Copy link
Copy Markdown
Contributor

Jenkins, retest this please.

@SparkQA

SparkQA commented Dec 1, 2015

Copy link
Copy Markdown

Test build #46938 has finished for PR 10039 at commit 81d973b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai

yhuai commented Dec 1, 2015

Copy link
Copy Markdown
Contributor

I think we need to delete the hive-exec jar from amp-jenkins-worker-03. The failure of https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46938/consoleFull was caused by a bad hive jar in the maven/ivy cache.

@JoshRosen

Copy link
Copy Markdown
Contributor

@yhuai which JAR specifically? I can log in to that machine and remove it.

@liancheng

Copy link
Copy Markdown
Contributor Author

@JoshRosen I guess it's safe to simply remove all Hive 1.2.1 and 1.2.1.spark jars in .m2 and .ivy2.

@liancheng

Copy link
Copy Markdown
Contributor Author

@yhuai Sent PR #10063 to backport this one to master.

asfgit pushed a commit that referenced this pull request Dec 1, 2015
This PR backports PR #10039 to master

Author: Cheng Lian <lian@databricks.com>

Closes #10063 from liancheng/spark-12046.doc-fix.master.
@marmbrus

marmbrus commented Dec 1, 2015

Copy link
Copy Markdown
Contributor

Can you close this now?

asfgit pushed a commit that referenced this pull request Dec 1, 2015
This PR backports PR #10039 to master

Author: Cheng Lian <lian@databricks.com>

Closes #10063 from liancheng/spark-12046.doc-fix.master.

(cherry picked from commit 69dbe6b)
Signed-off-by: Yin Huai <yhuai@databricks.com>
@yhuai

yhuai commented Dec 1, 2015

Copy link
Copy Markdown
Contributor

#10063 is in both master and branch 1.6. We can close this.

@liancheng liancheng closed this Dec 2, 2015
@liancheng liancheng deleted the spark-12046.doc-fix branch December 2, 2015 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants